Repository navigation
ci: clean-room-review gate counts only current-head, latest, write-access verdicts (sc-36322) - #4
Conversation
…cess verdicts (sc-36322) The SOC 2 CC8.1 merge gate (byte-identical in 85 repos, the only review control there) passed on stale, overridden and outsider reviews: 1. A clean-room APPROVE comment on an older commit still passed after new pushes (commit_id was never read). 2. Any APPROVED outweighed every CHANGES_REQUESTED, including a later one from the same reviewer. 3. The body regex matched "cannot APPROVE" and sub-reviewer APPROVEs inside a REQUEST_CHANGES body, and missed the real "Clean-room re-review" header. 4. Any account's review counted (39 of the repos are public). Now: each write/admin reviewer's latest verdict counts (permission read via the collaborators API, since author_association hides private members); a clean-room comment is judged by its first line only and blocks unless it reads "— **APPROVE**"; any block fails; a pass needs an approval on the head SHA. Non-default bases (beta-*, release) stay exempt. clean-room-review/test_gate.sh runs the workflow's own run: script against 27 fixture cases through a gh shim (old gate: 14 fail; new: 27/27), and is added to test.yaml's gate job. clean-room-review/rollout.sh copies the merged canonical file to the other 84 repos, one draft PR each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
inviscid
left a comment
There was a problem hiding this comment.
⛔ Clean-room review (tier L, 4 reviewers) — REQUEST_CHANGES
The new rule is sound: latest verdict per writer, header-only parsing, approval pinned to the head, outsider and read-only reviews dropped. The harness runs the shipped script; 27/27 pass and the old gate fails 14. One fail-open path remains.
Blocking (confirmed by an independent verifier):
.github/workflows/clean-room-review.yaml:42—gh api …/collaborators/$login/permission … || trueswallows every error, not only 404. If a CHANGES_REQUESTED reviewer's lookup hits a 5xx or rate limit, that reviewer is dropped and another writer's head approval passes (reproduced with a shim:blocking:[], exit 0).
Non-blocking:
:12—pull_requesttypes omitedited, so retargeting a PR from an exempt base to master keeps its exempt green check (pre-existing).:20— no per-PRconcurrencygroup; near-simultaneous review runs can race.:10-14— a PR is judged by its own edited copy of the gate (known; tamper follow-up).:42— the permission endpoint has not yet been exercised withGITHUB_TOKENin Actions.:55— header match is case-sensitive and loose; fails toward the documented format.clean-room-review/rollout.sh:38— an existing branch (422) aborts the whole sweep underset -e.clean-room-review/test_gate.sh:9-18— the shim defaults unknown users to "write" and has no lookup-failure case.- Design: any writer can dismiss another writer's block (GitHub-native; audit trail remains).
…n on base retarget (sc-36322) - A reviewer's permission lookup now tolerates only HTTP 404 (deleted or unknown account -> not a writer). Any other failure (5xx, rate limit) fails the gate instead of silently dropping that reviewer, which let a transient error on a blocker's lookup turn another writer's approval into a pass. - pull_request `edited` re-runs the gate when a PR's base is retargeted (e.g. an exempt beta-x PR moved to the default branch). - One concurrent run per PR (cancel-in-progress) so near-simultaneous review events cannot finish out of order. - Harness: the gh shim can fail a login's lookup with a given status; 4 new cases (502 blocker, 502 approver, 404 approval, 404 blocker). 31/31 pass; the previous workflow fails both 502 cases. - rollout.sh: an existing branch (422) skips the repo and is listed for a manual check; next steps spell out the head-pinned clean-room review each rollout PR needs before its gate turns green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
inviscid
left a comment
There was a problem hiding this comment.
✅ Clean-room review (tier L, round 2, 2 reviewers) — APPROVE
Round 1 (4 reviewers) blocked on one confirmed fail-open: the reviewer permission lookup swallowed every error, so a transient API failure dropped a blocking reviewer. d8397d1 fixes it (only HTTP 404 means "not a writer"; any other failure exits 1) with cases that fail on f1c5f35, and adds edited to the triggers, a per-PR concurrency group, and a rollout.sh skip for existing branches. Harness 31/31 runs the shipped script; the old gate fails 17. No fail-open path found: pipefail covers the reviews fetch, the permission loop reads from a redirect so its exit 1 ends the script, and every ${{ }} reaches the script through env. Live: run 42550399413 showed GITHUB_TOKEN can read the permission endpoint and the gate blocks on a clean-room REQUEST_CHANGES.
Non-blocking findings (recorded, not gating):
- .github/workflows/clean-room-review.yaml:22 — concurrency group keyed only on the PR number; an out-of-order review run for an old head can cancel the run for the new head, leaving it
cancelleduntil rerun (fails closed). Addinghead.shawould avoid it. - .github/workflows/clean-room-review.yaml:70 — header match is case-sensitive; a hand-typed
clean-room review — REQUEST_CHANGEScomment is ignored (native REQUEST_CHANGES still blocks; /self-approve always writes the canonical header). - .github/workflows/clean-room-review.yaml:82,85 —
[ "$(jq …)" -gt 0 ]in anifescapesset -e; unreachable today since verdict.json was just written. - .github/workflows/clean-room-review.yaml:13-15 — revoking an approver's write access does not re-run the gate.
- .github/workflows/clean-room-review.yaml:10-14 — a PR is judged by its own edited copy of the gate (tracked tamper follow-up).
- clean-room-review/rollout.sh:48 — a failed contents PUT aborts the sweep after the branch exists; a rerun skips it as EXISTS.
- clean-room-review/test_gate.sh:8 — undeclared
yq(mikefarah) dependency; preinstalled on ubuntu-latest.
Fixes sc-36322 (epic 36328). The SOC 2 CC8.1 merge gate, identical in 85 repos (blob
c4b002e0), passed on:commit_idcheck),New rule: take each write/admin reviewer's latest verdict. Permission comes from
collaborators/<login>/permission, becauseauthor_associationhides private members. A clean-room comment is judged by its first non-blank line, and anything other than— **APPROVE**blocks. Any block fails the gate. A pass needs an approval whosecommit_idis the head SHA (D1(a): native approvals too). Non-default bases (beta-*, release) stay exempt.Review round 1 (d8397d1):
|| truedropped that reviewer, so a transient error on a blocker's lookup could let another writer's approval pass.editedto thepull_requesttypes, so retargeting a PR's base (an exempt beta-x PR moved to master) re-runs the gate. This is trigger config, so the harness, which runs the step script only, does not cover it.concurrency: clean-room-review-<PR#>withcancel-in-progress, so near-simultaneous review events cannot finish out of order.rollout.sh: an existing branch (HTTP 422) now skips that repo and lists it under "Branch exists, check manually". The printed next steps now say each rollout PR needs a clean-room review pinned to its head commit before the gate turns green.Tests:
clean-room-review/test_gate.shruns the workflow's ownrun:script against 31 fixture cases through aghshim. A fixture can fail any login's permission lookup with a chosen status: 404 or 502. The original gate fails 14 of the first 27 cases. f1c5f35 fails both 502 cases. The new gate passes 31/31. The harness is added totest.yaml's gate job (renamed "Gate regression tests"; this repo requires onlyclean-room-review). Local ci-gate PASS on d8397d1. actionlint and shellcheck are clean.Live read-only check: I ran the new script against real reviews on 10 PRs merged on 2026-11-08. Every /self-approve record passes on its head: COMMENT on own PRs (infra#1397, plaid#8878/8879, docs#1209, xlwings#82) and APPROVE on others' (apps#514/515, sensor#14). infra#1386 passes only via the 19:35Z head re-review. beta-* RCs (plaid#8880, cp-rest#746) are exempt.
Rollout (after merge, not run):
clean-room-review/rollout.shcopies the merged canonical file to every repo whose default branch still holdsc4b002e0(84 today, surveyed), one draft PR each. UseDRY_RUN=1to survey only.Out of scope: /self-approve still needs to pin
commit_idand drop--admin(local skill, plan step 3). Also deferred: the tamper/reusable-workflow follow-up.🤖 Generated with Claude Code